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RELATIONSHIPS  AMONG  FACTORS  IN  NEW  OFFICER 
EFFECTIVENESS  REPORT  SYSTEM 


I.  INTRODUCTION 

In  1974,  a new  Air  Force  Officer  Effectiveness 
Report  (OER)  system  became  operational.  A 
significant  change  introduced  in  the  system  is  the 
controlled  distribution  of  Evaluation  of  Potential 
ratings  in  Block  V of  AF  Form  707.  The  con- 
trolled distribution  aspect  of  the  system  imposes  a 
limit  of  22%  of  the  Block  V ratings  given  by  a 
reviewer  in  the  top  block,  a limit  of  28%  in  the 
second  block,  and  the  option  to  distribute  the 
remaining  50%  across  the  bottom  four  blocks. 
These  controls  apply  only  to  reviewers.  The  first 
rating  official  in  the  chain,  the  rater,  and  the 
second,  the  additional  rater,  are  not  required  to 
conform  to  the  specified  distribution  in  assigning 
Block  V ratings.  Another  significant  departure 
from  the  old  system  is  a constraint  placed  upon 
the  rater  in  assigning  ratings  to  the  10  Performance 
Factors  in  Block  111  of  Form  707.  A rating  on  a 
Performance  Factor  may  range  from  Far  Below 
Standard  to  Well  Above  Standard.  This  range  is 
graduated  in  five  steps  with  Meets  Standard  at  the 
midpoint.  The  rater  also  has  the  option  of  using  a 
Not  Observed  or  Not  Relevant  rating,  if 
appropriate,  on  a specific  factor.  If  the  rater  does 
not  use  either  the  Meets  Standard  or  Not  Observed 
block,  he  must  document  the  rating  with  a specific 


example  of  performance.  The  28%  limitation  on 
second  block  ratings  of  Evaluation  of  Potential 
was  removed  in  1977. 

A statistical  analysis  has  been  carried  out  to 
focus  on  the  individual  Performance  Factors, 
Evaluation  of  Potential  ratings,  and  the  interrela- 
tionships among  these  variables.  Data  used  in  the 
study  are  the  ratings  contained  in  9,230  controlled 
reports  prepared  on  lieutenant  colonel  ratees 
during  the  window  period  30  November  1974  - 
31  March  1975.  These  reports  constitute  the  first 
controlled  report  cycle  in  the  new  OER  system. 


II.  PERFORMANCE  FACTORS 

For  the  purpose  of  this  study,  Performance 
Factor  ratings  were  assigned  numerical  values  from 
5 for  Well  Above  Standard  to  1 for  Far  Below 
Standard.  Not  Observed/Not  Relevant  ratings  were 
not  converted  to  this  numeric  scale.  Of  the  9.230 
reports.  9.149  had  complete  (numeric)  data  avail- 
able on  all  10  Performance  Factors.  The  means 
and  standard  deviations  of  ratings  assigned  by  the 
rater,  by  the  additional  rater,  and  by  the  reviewer 
are  displayed  in  Table  1. 


Table  1.  Performance  Factor  Means  and  Standard  Deviations 


Performance  Factor 

Ratar 

Additional 

Ratar 

Reviewer 

Maan 

SD 

Mean 

SO 

Mean 

SD 

Job  Knowledge 

4.67 

.60 

4.67 

.60 

4.67 

.60 

Judgment  and  Decisions 

4.47 

.71 

4.46 

.72 

4.46 

.72 

Plan  and  Organize  Work 

4.53 

.69 

4.52 

.69 

4.51 

.69 

Management  of  Resources 

4.52 

.70 

4.51 

.70 

4.51 

.70 

Leadership 

4.51 

.71 

4.50 

.72 

4.49 

.72 

Adaptability  to  Stress 

4.49 

.73 

4.48 

.73 

4.48 

.73 

Oral  Communication 

4.42 

.75 

4.41 

.75 

4.41 

.75 

Written  Communication 

4.46 

.73 

4.45 

.73 

4.45 

.73 

Professional  Qualities 

4.61 

.67 

4.61 

.67 

4.61 

.67 

Equal  Opportunity  Participation 

4.42 

.80 

4.41 

.80 

4.41 

.80 
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For  the  entire  group  of  9,230  reports,  the 
percent  of  ratings  assigned  each  value  by  raters  is 
shown  in  Table  2.  Percentages  are  rounded  to  the 
nearest  tenth. 

It  is  evident  that  the  predominant  factor  rating 
is  5,  Well  Above  Standard,  for  each  of  the  ten 
Performance  Factors.  The  Job  Knowledge  factor  is 
rated  Well  Above  Standard  more  frequently, 
74.1%,  than  any  other  factor.  Conversely,  it  is 
rated  Meets  Standard  less  frequently,  6.4%,  than 
any  other  factor.  Oral  Communication  is  rated 
Well  Above  Standard  by  raters  less  frequently, 
57.8%,  than  any  other  factor.  Equal  Opportunity 
Participation  is  rated  Meets  Standard  more 
frequently,  20.0%,  than  any  other  factor.  The 
number  of  Below  Standard  ratings  is  insignificant. 
It  varies  from  one  for  Equal  Opportunity 
Participation  to  20  for  the  Leadership  factor. 
There  is  a single  rating  of  Far  Below  Standard 
among  the  9,230  reports.  It  is  for  Professional 
Qualities.  The  number  of  Not  Observed  ratings 
ranges  from  zero  for  Equal  Opportunity  Participa- 
tion to  59  for  the  Management  of  Resources 
factor. 


The  degree  to  which  separate  rating  officials  in 
the  rating  chain  agree  in  the  use  of  Performance 
Factor  ratings  is  suggested  by  nearly  identical 
means  for  the  three  classes  of  rating  officials  on 
each  Performance  Factor.  This  agreement  is  also 
indicated  by  the  fact  that,  of  the  9,149  reports  for 
which  complete  data  are  available,  93.1%  contain  a 
set  of  Performance  Factor  ratings  assigned  by  the 
rater  which  is  identical  to  ratings  assigned  by  the 
additional  rater.  Also,  91.8%  of  the  reports 
contain  identical  ratings  assigned  by  the  rater  and 
by  the  reviewer.  The  additional  rater/reviewer 
agreement  is  98.5%. 

Correlation  coefficients  reflecting  the  rela- 
tionships between  Performance  Factors  were 
computed  for  the  9,149  reports  containing 
complete  data.  Major  results  of  this  correlational 
analysis  are  summarized  in  Table  3.  Data  are 
organized  separately  for  ratings  assigned  by  raters, 
by  additional  raters,  and  by  reviewers.  The  table 
displays  the  highest  and  the  lowest  correlation  of 
each  factor  with  the  remaining  factors  witliin  the 
ratings  assigned  by  each  type  of  rating  official.  The 
median  is  also  reported. 


Table  2.  Distribution  of  Performance  Factor  Ratings 

( Raters) 


Ratines 

Not 

Performance  Factor  5 4 3 2 1 Observed 


Job  Knowledge 

74.1% 

Judgment  and  Decisions 

59.9% 

Plan  and  Organize  Work 

63.7% 

Management  of  Resources 

63.3% 

Leadership 

63.3% 

Adaptability  to  Stress 

62.8% 

Oral  Communication 

57.8% 

Written  Communication 

60.1% 

Professional  Qualities 

71.5% 

Equal  Opportunity  Participation 

61.4% 

Total 

63.8% 

19.4% 

6.4% 

0.1% 

0.0% 

0.0% 

27.6% 

12.4% 

0.2% 

0.0% 

0.0% 

25.4% 

10.7% 

0.2% 

0.0% 

0.0% 

24.5% 

11.5% 

0.1% 

0.0% 

0.7% 

24.0% 

12.3% 

0.2% 

0.0% 

0.1% 

23.3% 

13.6% 

0.2% 

0.0% 

0.2% 

26.4% 

15.7% 

0.1% 

0.0% 

0.0% 

25.7% 

14.0% 

0.1% 

0.0% 

0.1% 

18.3% 

10.0% 

0.2% 

0.0% 

0.0% 

18.6% 

20.0% 

0.0% 

0.0% 

0.0% 

23.3% 

12.7% 

0.1% 

0.0% 

0.1% 
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Table  3.  Performance  Factor  Intercorrelations 


Pcrtormanc*  Factor 

Ratar 

Additional  Ratar 

R aviawar 

High 

Low 

Mdn. 

High 

Low 

Mdn. 

High 

Low 

Mdn. 

Job  Knowledge 

.60 

.44 

.54 

.59 

.43 

.53 

.58 

.42 

.53 

Judgment  and  Decisions 

.72 

.52 

.64 

.71 

.52 

.63 

.71 

.51 

.63 

Plan  and  Organize  Work 

.66 

.51 

.62 

.64 

.51 

.61 

.64 

.50 

.61 

Management  of  Resources 

.64 

.55 

.60 

.64 

.53 

.59 

.64 

.53 

.38 

Leadership 

.72 

.53 

.64 

.71 

.52 

.63 

.71 

.52 

.63 

Adaptability  to  Stress 

.71 

.55 

.63 

.70 

.55 

.62 

.70 

.54 

.61 

Oral  Communication 

.66 

.54 

.60 

.65 

.53 

.60 

.65 

.52 

.59 

Written  Communication 

.66 

.53 

.59 

.65 

.52 

.58 

.65 

.52 

.56 

Professional  Qualities 

.69 

.54 

.60 

.68 

.52 

.59 

.68 

.52 

.58 

Equal  Opportunity  Participation 

.56 

.44 

.55 

.55 

.43 

.54 

.55 

.42 

.53 

All  correlations  are  positive.  The  correlations 
among  ratings  of  Performance  Factors  assigned  by 
raters  range  from  .44  to  .72.  The  medians  range 
from  .54  to  .64.  Inspection  of  Table  3 indicates 
that  raters’  Performance  Factor  ratings  are 
moderately  correlated  within  all  ten  Performance 
Factors  and  with  minor  differentiation  between 
factors.  The  data  also  indicate  a nearly  identical 
range  of  correlations  within  the  ratings  of 
Performance  Factors  assigned  by  additional  raters 
as  well  as  those  assigned  by  reviewers. 

The  agreement  between  the  use  of  Performance 
Factor  ratings  among  the  three  levels  of  rating 


officials  is  shown  in  Table  4.  The  correlation 
coefficients  between  each  pair  of  rating  officials 
for  the  ratings  on  a given  factor  are  shown. 


In  general,  the  correlations  are  extremely  high 
and  indicate  that  only  in  rare  cases  do  additional 
raters  override  Performance  Factor  ratings  assigned 
by  raters.  The  correlations  between  the  ratings 
assigned  by  additional  raters  and  reviewers  are 
slightly,  but  systematically,  higher  than  those  for 
raters  and  additional  raters.  This  indicates  the  even 
less  frequent  override  of  a factor  rating  by  the 
reviewer. 


Table  4.  Correlation  Between  Rater  Categories 


Parformanca  Factor 

Ratar/ 

Additional 

Ratar 

Ratar/ 

R aviawar 

Additional 

Ratar/ 

Ravlawar 

Job  Knowledge 

.97 

.96 

.99 

Judgment  and  Decisions 

.97 

.96 

.99 

Plan  and  Organize  Work 

.97 

.96 

.99 

Management  of  Resources 

.97 

.96 

.99 

Leadership 

.96 

.96 

.99 

Adaptability  to  Stress 

.98 

.97 

.99 

Oral  Communication 

.97 

.97 

1.00 

Written  Communication 

.97 

.97 

1.00 

Professional  Qualities 

.97 

.97 

1.00 

Equal  Opportunity  Participation 

.98 

.97 

.99 

111.  PERFORMANCE  FACTORS  AND 
EVALUATION  OF  POTENTIAL 

In  this  section  the  relationships  between 
Performance  Factor  ratings  contained  in  Block  111 
of  the  Officer  Effectiveness  Report  and  ratings  of 
Evaluation  of  Potential  in  Block  V of  the  report 
will  be  examined.  As  background  for  this  discus- 
sion, summary  statistics  for  Block  V ratings  are 
presented  in  Table  5.  There  are  six  blocks  available 
to  rate  Evaluation  of  Potential.  The  controlled 
distribution  feature  of  the  system  imposes  a limit 
of  22%  of  each  reviewer’s  ratings  in  the  top  block 
and  a limit  of  50%  in  the  top  two  blocks.  For  the 
purpose  of  this  report,  numerical  values  1 through 
6 have  been  assigned  to  top  block,  second 
block, . . .sixth  block  ratings. 


Table  5.  Distribution  of  Ratings 
of  Evaluation  of  Potential 


Rating*  of 
Evaluation 
of  Potential 

Rater 

Additional 

Rater 

Reviewer 

i 

41.5% 

27.9% 

21.8% 

2 

33.7% 

32.7% 

29.0% 

3 

23.5% 

37.8% 

47.3% 

4 

1.2% 

1.5% 

1.8% 

5 

0.1% 

0.1% 

0.1% 

6 

0.0% 

0.0% 

0.0% 

The  controlled  distribution  target  percentages 
have  been  closely  adhered  to  across  the  population 
of  reviewers. 


The  extent  of  the  relationship  between 
individual  Performance  Factor  ratings  assigned  by 
raters  and  the  Evaluation  of  Potential  rating  is 
presented  in  Table  6.  The  first  column  displays  the 
validity  of  Block  III  ratings  assigned  by  the  rater 
for  the  rater’s  Evaluation  of  Potential  ratings,  the 
second  column  displays  the  validity  coefficients  of 
the  rater’s  Block  111  ratings  for  the  additional 
rater’s  Evaluation  of  Potential;  and  the  final 
column  contains  the  validity  coefficients  of  rater’s 
Block  III  ratings  for  the  reviewer’s  Evaluation  of 
Potential. 

All  validity  coefficients  are  negative,  since 
numerically  high  ratings  on  Performance  Factors 
tend  to  be  associated  with  numerically  low  ratings 
on  Evaluation  of  Potential.  In  displaying  validity 
coefficients  in  Table  6 and  in  the  discussion  which 
follows,  references  to  these  negative  correlation 
coefficients  will  be  omitted,  since  the  negative 
signs  are  an  artifact  of  the  scaling  procedure. 
Validity  coefficients  for  raters’  Evaluation  of 
Potential  ratings  range  from  .37  for  Equal 
Opportunity  Participation  to  .60  for  Leadership. 
Except  for  Equal  Opportunity  Participation,  the 
validity  coefficients  are  tightly  packed  in  the  range 
.47  to  .60.  The  same  pattern  of  validity 
coefficients  is  evident  for  both  additional  raters' 
and  reviewers’  Evaluations  of  Potential.  However, 
the  magnitude  of  the  validities  systematically 
decreases  from  the  raters’  to  the  additional  raters’ 
to  the  reviewers’  Evaluations  of  Potential.  The 
Leadership  factor  is  the  most  valid  for  Evaluations 
of  Potential  for  each  class  of  rating  official.  Equal 
Opportunity  Participation  is  the  least  valid  in 


Table  6.  Validity  of  Rater  Performance 
Factors  for  Block  V Ratings 

(Sign  Reversed) 


Performance  Factor 

Rater 

Additional 

Rater 

Reviewer 

Job  Knowledge 

.47 

38 

.34 

Judgment  and  Decisions 

.59 

.48 

.43 

Plan  and  Organize  Work 

.54 

.45 

.40 

Management  of  Resources 

.51 

.41 

.37 

Leadership 

.60 

.50 

.45 

Adaptability  to  Stress 

57 

.46 

.41 

Oral  Communication 

.53 

.44 

.40 

Written  Communication 

.50 

.40 

.36 

Professional  Qualities 

.52 

.43 

.39 

Equal  Opportunity  Participation 

.37 

.30 

.27 

i 
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each  class.  Similarly,  Judgment  and  Decisions  falls  The  Variable  Only  columns  in  Table  7 display 

next  to  Leadership  for  all  three  rating  officials,  the  squared  validity  coefficients  of  the  individual 

and  Job  Knowledge  falls  next  to  Equal  Opportu-  Performance  Factors.  To  illustrate  how  data  in 

nity  Participation  for  all  three  rating  officials.  Table  7 may  be  interpreted,  the  10  Performance 

To  examine  the  extent  to  which  combinations  Factors  as  a group  have  an  R2  of  .4540  for  raters’ 

of  Performance  Factors  account  for  ratings  of  Evaluation  of  Potential.  When  considered  alone, 
Evaluation  of  Potential,  several  series  of  multiple  Leadership  factor  has  a squared  validity  coeffi- 

regression  analyses  were  performed.  Using  the  c'ent  -3618.  When  Leadership  is  removed  from 
raters’  Evaluation  of  Potential  as  the  dependent  861  °f  10  Performance  Factors,  the  remaining 

variable,  the  squared  multiple  correlation  coeffi-  n*ne  Performance  Factors  yield  an  R2  of  .4408. 

cient.R2  of  the  10  Performance  Factors  is  .4540.  Individual  Performance  Factors  account  for 

This  means  that  an  optimally  weighted  linear  between  14%  and  36%  of  the  variability  in  raters 

combination  of  Performance  Factor  ratings  Evaluation  of  Potential.  However,  none  of  the  10 

assigned  by  raters  can  account  for  45%  of  the  independent  contributions  is  of  practical 

variability  within  raters’  Evaluation  of  Potential.  importance.  These  individual  independent  con- 

The  corresponding  R2  values  are  .3036  and  .2438  tributions  range  from  .0007  for  Written  Com- 

for  additional  raters’ and  reviewers’ Evaluations  of  munication  (.4540  to  .4533)  to  .0132  for 

Potential,  respectively,  using  raters’  Performance  Leadership  (.4520  to  .4408).  A similar  pattern 

Factor  ratings  as  independent  variables.  In  the  first  emerges  when  comparable  data  for  additional 

series  of  10  multiple  regression  problems,  each  of  rater  reviewers  Evaluations  of  Potential  are 
the  10  Performance  Factors  was  removed.  The  examined.  Although  individual  Performance 

comparison  of  the  resulting  R2  with  the  full  model  Factors  differ  somewhat  in  their  predictability  of 

R2  using  all  10  Performance  Factors  as  indepen-  additional  raters'  and  reviewers’  Evaluations  of 

dent  variables  indicates  the  extent  of  the  indepen-  Potential,  no  single  Performance  Factor  makes  a 

dent  contribution  of  each  Performance  Factor  to  substantial  independent  contribution  to  the  pre- 

the  remaining  nine  Performance  Factors.  A similar  disability  of  these  two  dependent  variables, 

series  of  10  regression  systems  was  computed  using  A distinct  decrease  in  ‘he  predictability  of  the 

additional  raters’  Evaluation  of  Potential  as  the  dependent  variable  is  observed  between  the  raters’, 

dependent  variable  and  deleting  one  of  the  10  additional  raters’,  and  reviewers’  Evaluations  of 

raters'  Performance  Factors  in  each  of  the  Potential.  In  order  to  further  ascertain  and 

problems.  Finally,  a third  series  was  carried  out  evaluate  the  role  of  raters’  Performance  Factor 

using  the  reviewers’  Evaluation  of  Potential  as  the  ratings  in  the  assignment  of  reviewer  ratings  of 

dependent  variable.  A summary  of  these  results  is  potential  under  the  limitations  of  the  constraints 

displayed  in  Table  7.  imposed  by  the  controlled  distribution  system. 


Table  7.  Predictability  of  Evaluation  of  Potential 


Rater  Additional  Ratar  Reviewer 


Performance  Factor 

Var. 

Only 

Var. 

Delated 

Var. 

Only 

Var. 

Delated 

Var. 

Only 

Var. 

Deleted 

Job  Knowledge 

.2229 

.4529 

.1425 

3033 

.1145 

.2464 

Judgement  and  Decisions 

.3506 

.4451 

.2343 

.2971 

.1879 

.2420 

Plan  and  Organize  Work 

.2940 

.4506 

.1995 

.3001 

.1616 

.2439 

Management  of  Resources 

.2625 

.4523 

.1694 

.3030 

.1385 

.2460 

Leadership 

.3618 

.4408 

.2450 

.2930 

.2009 

.2373 

Adaptability  to  Stress 

.3204 

.4492 

.2105 

.3009 

.1670 

.2452 

Oral  Communication 

.2831 

.4497 

.1921 

.2996 

.1583 

.2430 

Written  Communication 

.2459 

.4533 

.1586 

.3034 

.1314 

.2463 

Professional  Qualities 

.2758 

.4524 

.1822 

.3027 

.1513 

.24  5 S 

Equal  Opportunity  Participation 

.1379 

.4487 

.0907 

.3000 

.0750 

.2438 

i 


i 
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additional  multiple  regression  systems  were 
computed.  For  a model  in  which  the  reviewer’s 
Evaluation  of  Potential  was  the  dependent  variable 
and  the  independent  variables  consisted  of  both 
the  rater’s  and  additional  rater’s  Evaluations  of 
Potential,  as  well  as  the  10  Performance  Factor 
ratings  assigned  by  the  rater,  the  R2  is  .7602. 
When  only  the  ratings  of  Evaluation  of  Potential 
assigned  by  the  rater  and  by  the  additional  rater 
are  used  to  predict  the  reviewer’s  Evaluation  of 
Potential,  R2  = .7599.  It  is  evident  that  raters’ 
Performance  Factor  ratings  have  virtually  no 
systematic  relationship  to  reviewers’  Evaluations 
of  Potential  after  the  Evaluations  of  Potential 
assigned  by  the  rater  and  the  additional  rater  have 
been  taken  into  account.  The  validity  of  the 
additional  raters’  Evaluation  of  Potential  for  the 
reviewers’  Evaluation  of  Potential  is  .87,  and  the 
squared  validity  is  .7592.  The  validity  of  the 
raters’  Evaluation  of  Potential  is  .69,  and  the 
squared  validity  is  .4713.  While  both  Evaluation  of 
Potential  variables  (rater  and  additional  rater)  have 
substantial  validities  for  the  reviewers’  Evaluations 
of  Potential,  and  both  variables  individually 
account  for  a larger  proportion  of  the  variability  in 
reviewers’  Evaluation  of  Potential  than  a linear 
combination  of  all  10  Performance  Factors,  it  is 
clear  that  the  additional  raters’  Evaluation  of 
Potential  accounts  for  substantially  all  of  the 
predictable  variance  in  the  reviewers’  Evaluation  of 
Potential.  This  fact  is  further  substantiated  by  the 
agreement  between  additional  raters  and  reviewers 
in  assigning  Evaluation  of  Potential  ratings.  On 
84.4%  of  the  reports  the  additional  rater  and 
reviewer  assigned  identical  ratings;  in  15.1%  the 
reviewer  lowered  the  additional  rater’s  rating 
(higher  numerical  value  on  a scale  of  l-6);and  in 
0.6%  the  reviewer  raised  the  rating.  Corresponding 
figures  for  the  agreement  between  reviewers  and 
raters  are  60.3%,  38.4%,  and  1.3%,  respectively. 

When  the  rater’s  Evaluation  of  Potential  and  Iris 
ratings  of  Performance  Factors  are  used  in  a 
multiple  regression  system  to  account  for  the 
additional  rater’s  Evaluations  of  Potential,  R2  = 
.5929.  As  noted  previously,  the  squared  multiple 
correlation  coefficient  of  10  Performance  Factors 
for  the  additional  rater’s  Evaluation  of  Potential  is 
.3036.  The  validity  of  the  rater’s  Evaluation  of 
Potential  for  the  additional  rater’s  Evaluation  of 
Potential  is  .77,  and  the  squared  validity  is  .5903. 
Although  both  Performance  Factor  ratings  and 


rater’s  Evaluation  of  Potential  account  for  a 
substantial  proportion  of  the  variability  in  the 
additional  rater’s  Evaluation  of  Potential,  the 
independent  contribution  attributable  to  the 
rater’s  Evaluation  of  Potential  is  considerably 
larger  than  that  of  the  Performance  Factors. 


IV.  DISCUSSION 

The  majority  of  the  Performance  Factor  ratings 
on  the  new  Form  707  are  high.  Mean  values  for 
individual  factors  all  exceed  4.4  on  a five-point 
scale.  There  is  virtually  no  use  of  the  lower  half  of 
the  scale.  Only  0.2%,  or  fewer,  of  the  reports 
contained  ratings  of  2 (Below  Standard)  on  any 
Performance  Factor.  The  Far  Below  Standard 
block  was  used  on  only  one  report  for  a single 
factor.  Apparently,  raters  see  no  particular 
problem  in  providing  specific  examples  of  Above 
Standard  and  Well  Above  Standard  performance, 
since  these  blocks  are  used  in  23%  and  63%  of  the 
ratings,  respectively.  An  analysis  of  the  content  of 
specific  examples  cited  by  raters  is  beyond  the 
scope  of  the  present  study.  There  would  seem  to 
be  some  question  as  to  whether  specific  examples 
of  Above  Standard  and  Well  Above  Standard 
performance  cited  by  raters  are  of  uniform 
quality. 

In  general,  second-  and  third-level  raters  in  the 
rating  chain  accept  ratings  of  performance 
provided  by  the  rater.  There  was  complete  agree- 
ment in  Performance  Factor  ratings  between  raters 
and  additional  raters  in  more  than  93%  of  all 
reports.  Over  92%  of  the  reports  reflected 
complete  agreement  between  the  raters  and 
reviewers.  Over  98%  of  all  reports  reflect  complete 
agreement  between  additional  raters  and  reviewers. 
The  stability  of  Performance  Factor  mean  values 
across  categories  of  rating  officials  and  the 
extremely  high  correlation  coefficients  between 
each  pair  of  types  of  rating  officials  for  each 
Performance  Factor  are  further  evidence  of  the 
extent  of  agreement.  As  might  be  expected,  there 
is  slightly  greater  agreement  betweeen  additional 
raters  and  reviewers  than  between  raters  and 
additional  raters.  Agreement  between  raters  and 
reviewers,  although  high,  is  slightly  below  the  level 
of  agreement  between  raters  and  additional  raters. 

Intercorrelations  among  Performance  Factor 
ratings  are  all  positive  and  of  moderate  degree. 
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Equal  Opportunity  Participation  is,  in  general,  less 
highly  correlated  with  the  remaining  factors  than 
is  any  other  single  factor. 

Performance  Factors  taken  in  combination 
correlate  more  highly  with  Evaluation  of  Potential 
ratings  than  do  individual  Performance  Factors. 
Although  raters’  Performance  Factor  ratings  in 
combination  account  for  approximately  45%  of 
the  variability  in  raters’  Evaluation  of  Potential, 
the  percent  of  variance  accounted  for  is  reduced  to 
30%  for  additional  raters’  Evaluation  of  Potential 
and  to  24%  for  reviewers’  Evaluation  of  Potential. 
No  single  Performance  Factor  makes  an  appreci- 
able independent  contribution  to  these  levels  of 
predictive  efficiency.  The  statistical  analysis  of  the 
data  indicates  that  one  or  more  of  the  Perform- 
ance Factors  could  be  deleted  from  the  Form  707 
without  affecting  the  manner  in  which  the  rating 
official  evaluates  the  ratee’s  potential.  Com- 
parisons involving  the  independent  contribution  of 


each  of  the  Performance  Factors  indicate  that  it 
would  make  very  little  difference  which  Perform- 
ance Factor  or  subset  of  Performance  Factors  are 
removed  from  consideration.  Within  the  minor 
differences  that  do  appear,  the  Written  Com- 
munication and  Job  Knowledge  factors  make  less 
independent  contribution  than  do  the  other 
Performance  Factors. 

While  raters’  Performance  Factors  do  account 
in  some  degree  for  reviewers’  Evalution  of 
Potential  ratings,  the  best  single  predictor  of 
reviewers’  Evaluation  of  Potential  is  the  additional 
raters’  Evaluation  of  Potential.  Performance 
Factor  ratings  add  essentially  nothing  to  the 
predictive  efficiency  provided  by  the  additional 
raters’  Evaluation  of  Potential. 

The  above  discussion  pertains  to  lieutenant 
colonel  reports.  The  use  of  Performance  Factors 
and  the  relationship  of  Performance  Factors  for 
ratings  of  Evaluation  of  Potential  may  vary  con- 
siderably as  a function  of  the  ratee’s  grade. 


